Unsupervised deduplication using cross-field dependencies
Rob Hall
Charles Sutton
Andrew McCallum
Recent work in deduplication has shown that collective deduplication of different attribute types can improve performance. But although these techniques <i>cluster</i> the attributes collectively, they do not <i>model</i> them collectively. For example, in citations in the research literature, canonical venue strings and title strings are dependent -- because venues tend to focus on a few research areas -- but this dependence is not modeled by current unsupervised techniques. We call this dependence between fields in a record a <i>cross-field dependence</i>. In this paper, we present an unsupervised generative model for the deduplication problem that explicitly models cross-field dependence. Our model uses a single set of latent variables to control two disparate clustering models: a Dirichlet-multinomial model over titles, and a non-exchangeable string-edit model over venues. We show that modeling cross-field dependence yields a substantial improvement in performance -- a 58% reduction in error over a standard Dirichlet process mixture. 
